Top 7 Performance Problems in .NET Applications and How to solve them
Microsoft .NET Framework is one of the most popular application development platforms and programming languages. C# and ASP.NET frameworks are used by millions of developers for building Windows client applications, XML Web services, distributed components, client-server applications, database applications, and so on. It’s no surprise that ensuring top-notch performance of .NET applications is a foremost need for most application owners and developers.
There can be numerous reasons why .NET performance issues occur and cause applications to be slow. These include incorrect memory sizing, GC pauses, code-level errors, excessive logging of exceptions, high usage of synchronized blocks, IIS server bottlenecks, and so on. In this blog, we will look at some of the top performance problems in .NET applications and provide tips to troubleshoot and resolve them.
Top 7 .NET Application Performance Problems |
|---|
#1 Exceptions and Logs One Too Many
.NET exceptions are not a bad thing. Only errors are bad. This is what most developers believe. And it is true if exceptions are properly handled, i.e., thrown, caught and addressed (and NOT ignored). Too many cooks spoil the broth and it’s the same with exceptions. Too many unhandled exceptions can make the code inefficient and affect application performance. Hidden exceptions are worse: a minefield. When left unchecked, they can affect web page load times.
Another .NET problem is excessively logging the exceptions. Logging could be a great tool in your debugging arsenal to identify abnormalities recorded at the time of application processing. But when logging is set up to catch exceptions at every tier of the application architecture, one could end up having the same exception logged at the web, service and data tiers. This could add additional load to the application code and increase response time. In production environments, one needs to be careful to only log fatal events and errors. Logging everything including informational messages, debugs and warnings can easily bloat your production log file and in turn affect code processing.
Helpful Troubleshooting Tips:
- Make sure your C# code has “try catch finally” blocks to handle exceptions.
- Leverage exception filters available in C# 6 and above, which allows specifying a conditional clause for each catch block
- Check for null values and use TryParse to avoid potential exceptions.
- Pay attention to second-chance exceptions, as these indicate that the first-chance exception came up and it wasn’t handled properly.
- Use exception handling and logging libraries such as Enterprise Library, NLog, Serilog, or log4net to log exceptions to a file or a database.
- Make sure you log exceptions only as much as needed and not end up bloating the log file.
#2 Overuse of Thread Synchronization and Locking
The .NET Framework offers many thread synchronization options such as inter-process mutexes, Reader/Writer locks, etc. There will be times when a .NET developer would write the code in such a way that only one thread could get serviced at a given time and the other parallel threads coming in for processing will have to wait in a queue. For example, a checkout application, according to its business logic, should process items one request at a time. Synchronization and locking help serialize the incoming threads for execution. By creating a synchronized block of code and applying a lock on a specific object, an incoming thread is required to wait until the lock on the synchronized object is available. While this strategy helps in certain situations, it should not be overused. Too much serialization of threads will increase wait time of incoming threads and end up slowing down user transactions.
Helpful Troubleshooting Tips:
- Use synchronized coding and locks only when necessary. Understand the need of the code execution before deciding to use locks.
- Scope the duration of locks optimally so that they are acquired late and released early and does not hold other threads in wait for a long time.
- To reduce concurrency issues, consider using loose coupling. Event-delegation models can also be used to minimize lock contention.
- Monitor the .NET code using code profiling tools to identify if thread locks are causing slow application processing.
#3 The Dreadful Application Hangs
When a specific URL is slow, it is one thing to handle. But when the IIS web site just hangs and all or most web pages take forever to load, it couldn’t get any worse. Typically, when an application is overloaded or deadlocked, a hang could occur. There are two types of application hang scenarios that .NET applications usually encounter.
A hard hang (IIS issue): This usually happens at the beginning of the request processing pipeline – where the request is queued. Because of an application deadlock, all available threads could get blocked, causing subsequent incoming requests to end up in a queue waiting to be serviced. This can also happen when the number of active requests exceed the concurrency limit configured on the IIS Server. Such a hang would manifest as requests getting timed out and receiving 503 Service Unavailable errors. A hard hang affects all URLs and the whole web application itself.
Helpful Troubleshooting Tips:
- Constantly track number of requests in queue in the IIS Server (Http Service Request Queues\ArrivalRate in Windows performance monitor). This should never exceed the request processing limit configured for the worker process.
- Also track how long requests are waiting in queues (Http Service Request Queues\MaxQueueItemAge in Windows performance monitor). This will help detect if the application is facing a potential hang.
- Also watch out for service unavailable and connection timeout errors by monitoring IIS server events.
A soft hang (ASP.NET issue): This usually happens due to a bad application code in a specific segment, impacting only a few URLs and not the full website. Typically, a hang caused by the ASP.NET controller or page happens at the ExecuteRequestHandler stage. To confirm this, you might want to break out a debugger to see exactly where the request is stuck. Check the module name, stage name and URL. The URL will indicate the controller/page causing the hang.
Helpful Troubleshooting Tips:
- Verify whether IIS is the issue or not by checking the Http Service Request Queues\CurrentQueueSize counter in the Windows performance monitor. If it’s 0, then there is no request stuck in an IIS queue.
- If it’s not an IIS issue, it must be a code-level problem in ASP.NET controller/page.
- Identify which URL(s) are hanging and get a detailed request trace using any code profiling. Verify the module name and stage name at which the request hangs to confirm it’s an ASP.NET issue.
- Code profiling using a transaction tracing tool can help identify the exact line of code where the problem exists.
#4 Frequent Garbage Collection Pauses
Garbage collection (GC) in .NET CLR is initialized when the memory used by the allocated objects on the managed heap exceeds the accepted threshold configured by the application developer. This is when the GC.Collect method jumps into action and reclaims the memory occupied by dead objects. GC in the CLR usually happens in Generation 0 heap where short-lived objects are stored. It is called Full GC when GC happens in Generation 2 heap, where long-lived objects are contained. Every time GC happens it adds a lot of CPU load on the CLR and slows down application processing. So, in the event of longer and more frequent GC pauses, the application tends to suffer slowdowns.
Helpful Troubleshooting Tips:
- Size GC heap memory properly and make sure GC limits are set as required.
- Avoid using objects and large strings where they are not needed.
- Track instances of GC, time taken for GC, and % of GC time spent by the JVM.
- Look for times when Full GC happens. This can cause application slowness.
- Judiciously use server GC or workstation GC based on application needs.
- Monitor the CLR layer end to end to identify memory usage, GC activity, CPU spikes, etc.
#5 IIS Server Bottlenecks
Microsoft IIS Server is a critical part of the .NET Framework. IIS is the web server which hosts the web application or web site built on .NET and runs W3WP process that is responsible to respond to the incoming requests. IIS also incorporates the Common Language Runtime (CLR), which is responsible to mete out resources for thread processing. Because IIS has various moving parts, a bottleneck in IIS could have a direct negative impact on the .NET application performance.
Commonly faced IIS Server problems:
- Server overload due to overutilization of resources such as memory, CPU, etc.
- High concurrent connections and connection drops
- Application pool failure
- Expiry of SSL certificates
- High response time of the ASP.NET request handling service
- High CLR wait time
- Improper caching
- HTTP errors including static and dynamic content errors and connection errors
Helpful Troubleshooting Tips:
- Right-size the IIS server so there’s no resource contention or overutilization of resources.
- Load balance with more IIS servers based on rate of incoming requests.
- Track SSL certificate validity and get alerted proactively before a certificate expires.
- Monitor all aspects of IIS performance, application pools, web sites and identify improper configuration and performance deviations.
#6 Slow Database Calls
It’s not always a .NET code issue that’s affecting application performance. Slow-running queries are often a common cause. But it’s usually the .NET application developers who get blamed for slow application performance. The reason for this is there’s no contextual visibility of how SQL performance affects .NET application processing. ADO.NET and ODP.NET connectivity issues could be one reason for query processing slowness, but the common reason is that the queries are not well-formed. Improper execution plans, missing indexes, poorly designed schema, small buffer pools, missing joins, improper caching, connections not being pooled properly, etc. are also reasons why query processing by the database could be affected.
While the DBAs are responsible for the database performance and query creation, the .NET application owner needs to track down query-level issues during application processing. This will help distinguish between code-level and database problems and not have the .NET developers spend cycles looking for issues in the code.
Helpful Troubleshooting Tips:
- Monitor query processing in context of application transactions to identify slow queries.
- Plan database sizing and configuration properly to ensure the consistent performance.
- Use database monitoring tools to identify and fix missing indexes, optimize the database layout by re-indexing, etc.
- Track database connectivity with the application to isolate any connection issues.
BONUS TIP: In addition to slow database calls, there could also be slowness due to external calls, such as HTTP, Web Service, WCF. .NET code profiling will help catch .NET performance problems cause by method-level issues, database query-level problems and slow remote procedure calls.
#7 Infrastructure at Fault: Not a .NET Problem, But Still a Problem for .NET!
.NET Framework is not a standalone tier. An application using .NET Framework will have many dependencies with the underlying infrastructure, such as any virtualized servers, containers or cloud infrastructure. Then, there could be backend storage devices. While these are not .NET problems directly, but a problem in any of these infrastructure components could affect .NET performance just the same.
Just like how we saw IIS Servers and database could have bottlenecks, a VM could be running out of resources, a SAN array could be experience high IOPS that it cannot handle, or if the .NET application is hosted on Azure there could be an App Service that is not running properly.
Network-related complaints top the chart in most application environments. There is always the blame game between whether it’s a network issue or the application issue. Network congestions, packet drops, or device failures could impact application performance and connectivity.
Helpful Troubleshooting Tips:
Total performance assurance of the .NET application environment requires correlated visibility of dependencies between the application and the supporting infrastructure. Make sure you implement a converged application and infrastructure monitoring strategy to catch infrastructure issues.
While you focus on catching and troubleshooting all these issues, it is also important to keep in mind that writing clean and efficient code solves many problems on the .NET side. Write good code, keep your systems and infrastructure in good health, and implement necessary tools for monitoring automation. This will help you deliver high-performing .NET applications and digital experience.
eG Enterprise is an Observability solution for Modern IT. Monitor digital workspaces,
web applications, SaaS services, cloud and containers from a single pane of glass.
